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A PROPOSED dl;sum wk train 



JiDUCATlOWAI. EVALUATORS AND DLW.LOPLRS 



In this third technical paper, the design for a training program for 
educational developers arid evaluators is explicated. Mentioned only in pass- 
ing are (I) the procedures by which die design i ;as generated and (2) the 
salient charaetcristi cs of the sewra l agencies that make up the corisortium; 
the reader interested in these eler rents is referred to Ted mi cal Papers 
Numbers l and 2 , respectively, in this series. 

The topics considered in this section include the folioring: 

L) A general description of the highiignts of the proposed training, 
x) The basic assumptions underlying the training program. 

3) The objectives of the training program. 

4) The eJenents of the three-level training program (Ph.D program, MA progrcim, 
a; id intensive training institutes) . 

3) l'l :c staff who will to associated with the program. 

6) The recruitment ana selection of trainees. 

7) Tlic i ru tenia Is development proposed. 

8) The evaluation plan for the training program. 

9) The* administrative*, monitoring, and cost accounting elements of the 
L mining system. 
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1 ■ General Description of the Trainin g Program. 

Generally, a multi -faceted training program is being proposed. TVo 
ole rants concern training leading to graduate level degrees at both the 
ires tors and doctorate level. A third element involves intensive training 
institutes, primarily for personnel already cn-the-job. The training 
program involves over a dozen consortium agencies, as was detailed in 
Technical Paper Number 2. 

Training leading to graduate degrees is proposed in four areas: the 

masters degree in evaluation; the rasters degree in development; the 
doctorate in development; and tloe doctors to in evaluation. Specific con- 
tont and planned e^ieriences for each of these degrees is explicated in 
Section IV bo lew. These activities include a wide variety of exceptional 
internship opportunities . Itecruitment will be national in scope, and 
screening will be intensive (see Section VT below) . 

The intensive training institutes in evaluation and development will 
te held in increments of short time duration (usually varying in length 
between one and four weeks) . They will be individually designed and 
tailored to reet the needs of the particular group of trainees that is 
oaring for the training (sec Section IV) . An experiment to determine 
flic effectiveness of various follow-up models is being proposed (Section 
VIII) . In regard to the training program, especially the short-term 
institutes, considerable rate rials development is oonterrplatod {Section 



It can be: noted in the written documents associated with designing 
the project that there has been an extensive shift in orientation over 
the six-rrnnth design period. This shift mi. gilt test be described as away 
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fran the traditional uni wr si tv -oriented model to a iisor -oriented 
model, with elaborate provisions for rreoning ful internship experiences 
for pro “Service trainees and high-pewered short -term institutes 
for two types of personnel: those already holding positions with 

evaluation and development responsibilities and those whose responsi- 
bilities are m the process of being redefined and will include 
uQwlopnent and evaluation. Addi tonally, many meaningful and ooirrnun- 
icative bridges are oontenplatcd linking the various training pro- 
grams pressed under this project., as well as linking them to existing 
training programs at Colorado and at many of the consortium units. 

Ml elements of the training program desiejned are considered 
in detail in the sections that follow i_n this technical paper. 
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1 1 . Ccnoj\fl_ Assumpt ions Unde lying t he Trainin g Program 

Ik. lew are discussed several assumptions, each of which has had 

a direct influence on the design of tiie proposed training programs. 

Supporting evidence for the assumptions ranges from hunches to relatively 

hard data and is indicated as cadi assumption is presented. It is 

encouraging to the developers of the design that each assumption 

does appear' to be support iblo , at least in large part. 

1) 'j hoie is a need for pre-sorvioe, graduate level training in 
evaluation and development. 

Supporting this assumption are a nurroer of extant iranpcwar studios. 
Most i^oont and relevant for the design project have teen two studies 
completed by the AfTRA Task Force on Res card- Training. The first 
concerned analyses of AKRA placement data for the years 1968, 1969, 
and 1970 (referred to initially in Technical Paper Number 1 in this 
series) . A definite need for new entrants into the fields of evaluation 
and development was apparent over that time period, although it was 
somewhat less pronounced recently with the advent of constricted 
federal funding. The second AHFft task force ot.xiy of fifty-eight 
erp layer's of P, D, D, and E personnel resulted in similar findings. 

Information collected during the course of the design grant 
confirms this need. M*>st institutions in this consortium, probably 
representing loss than one percent of the potential employment spectrum 
in R, I:, D, and E, expressed needs for additional personnel in evaluation 
and development , especially at the master's level. Possibly even a 
greater need identified was the need for addi tional training for 
present uiployces (see the working papers of the oormortiixn units, 
Appendices A - G, in Technical Paper Number ?) . 
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Data collected via questionnaire (slightly o^r fifty percent 
returns) resulted in compatible indications ( the actual questionnaire 
was presented in Appendix B in Tedinical Paper Number 1) . Results 
for evaluation are presented in Table 1; in Table 2, development data 
an: presented-* Hie tables are self-explanatory; and trends are apparent. 
Evan witJi no funding increase, about forty percent of the agencies 
intend to hire staff in both evaluation and development. With a 
twenty percent funding increase, the intention- to-h ire figure j unps to 
over eighty percent of the organizations. Mote, also, that saveral types 
of agencies plan to hire more than one person at each level, and that 
the greatest demand seems to lx: at tire master’s level (for bo fit 
evaluators and developers) . 

2 ) Iheru dees not exist, at this time, oven a moderate number of 
institutions that provide training in development. 

Project staff had inclinations to belie ve this assumption even 
before the design project caretenoed . As part of tie project, it v?as 
possible to tab: to many persons from centers and regional laboratories. 
Tliey reinforced this opinion. The national ciu-riculum orojects also 
made it clear in t! le i r work i ng pape r s ( see ap^ond i . ces in Techni ca 1 
Paper Nuirijor 2) tJ:at there is no single training institution to which 
they go when hiring r.e w staff. Rather, they d: ten line the skills of the 
particular role assignment that they hspe to fill and then conduct an 
elaborate icardi, often nationwide, to find such a person. 



* As over forty percent of the sampled agencies did not respond, the 
lx:roentages in Tallies 1, 2, and 3 may be inflated and should be 
mt erpnocod with caution. That is, many organizations electing rx>t 
to return tiie questionnaire iray have considered the training irrelevant; 
and/or not feasible. 
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It. also can be noted in passing that there is only a limited 
nuirbor of organizations that claim to have formal training progrcuns 
in evaluation. In the past, rrany persons graduating from research 
training programs have picked up assorted evaluation skills in the 
process. The various conceptual papers recently written should facilitate 
the process oi detail big the specifics of the research, development, 
diffusion, and evaluation domains? it is possible that much done 
previously in the name of "research" actually pertained more to 
evaluation than to research, 

3) There is a jumble of literature dealing with diffusion and 
change agents, but the existence of bard data in this area is 
rare . 

like many of the other design projects, personnel connected 
with this consortium a valiant effort to oonceptualize what 
training in diffusion woid be like. There is no question but 
that certain of the skills considered important, particularly by 
personnel in the national curriculum projects, clearly are critical 
for di ffusion. Indeed, the relationships between development, eval ration , 
and diffusion appear to be extensive. 

be that as it may, it is extremely difficult to locate and study 
dcxt'i v baling with diffusion, let alone extracting frem the data the 
principles upon which a training program for diffusers mi^it be 
based. Although the literature on suen roles as change agent, diffuser, 
and tiie like is abundant , evidence and specifics about effective 
metivodjlogies for them to use and critical ingredients of their training 
are essentially unknown. Being blessed with normal vision, the 
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preparers of this proposal were aware of the vacuum existing in the 
diffusion area and recognized the natural inclination to rush into 
that uninhabited space in order to it- crease tne likelihood of funding. 

The prefect staff and the censor ti um agencies, in this case, have 
successfully defended against that inclination; there are no 
plans that this prog rain train diffusers, l -.iuraily , there are dif t jl» ion- 
re la Led skills that are a part of evaluation and especially, de ve lorirent . 
Exanples of such skills are the use of ncclia, the use of ERIC, 

^nowLxlge of professional orcanizat ions and modes of canuunicaticr: 
they use (e.g., annual conventions), and the ability to oocrrruiiicat 
in writing. In the programs proposed herein, these activities and 
ski 1 Is are 'dewed as supporting, rather than central, in nature. 

11 lore fore , in this program, training in diffusion skills v/ill 

\x 2 incidental rather than planned. It should be noted, however, that 

our inability to rake advancements in the diffusion area will make us 

doth ly alert and prenu to examine closely those designs that propose 

to train dif fusers. Such efforts should lx* scrutinized to ascertain 

either tine brilliance of the designers or their effrontery. 

A) 1 here is a need for intensive training institutes in evaluation 
and dc\^lcprrrit for poisonne! whose roles call for skills in 
these areas, or for those whose roles likely will be redefined 
as a result of such training. 

Extensive input suggesting that this was a valid assumption first 
occurred in cho working papers submitted by the o>iisortiuti units 
i sej a; indices to Technical Paper Nurrber 2) and also from our disci is s ion: 
with onsorciuu agencies that did not rurxr.Lt working papers. Jn 
part piobobly explained by the current financial sq »?eze, many 
agencies were la iking "to mkc do" with their present personnel. 
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A53 a result, they haw pronounced needs for various types of on-the-jcb 
or of f-the- jcb training to either develop new skills in errployees or 
refurbish existing skills in certain evaluation and development areas. 

It also can be noted that job redefinition is needed in many cases. 
Institutes such as those planned should add to the pool of trained 
iampowor in evaluation and deve lcpiiient . The widespread consensus 
on the need for training institutes has eec i elected in the substantial 
level of resources ptoposed for that purpose in this design; institute- 
training clearly represents a substantial departure from what was 
inferred in our original response to RFP 70-1 2 . 

Responses on the questionnaires that were sent out reinforced 
our feelings in this rega:d; data are presented in Table 3. It can 
be noted that approxinvatoly 95 percent of the agencies returning 
questionnaires indicated interest in sending present enployees to 
evaluation institutes; the corresponding figure for development 
institutes was 96 percent. The averages presented in the table high- 
] j uh t tiie fact that most agencies were planning to send more than 
u single staff member. Recall that organizations would continue to 
pay the individual his regular salary while he attended the institute. 
This pronounced interest on the part of user organizations is responded 
to, and reflected in, the proposed training program. Particularly 
encouraging is that fact that the institutes, if conducted at an 
intense enough iewl, can serve a function similar to bringing now 
entrants into the fields of evaluation and development. That is, 
the skills imparted to trainees will allow them to undertake, quite 
possibly for the first time, responsibilities that are heavily evaluative 
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and developmental in nature; their job descriptions can be altered 
to reflect their new capabilities and, in effect, they become part 
of the E and D manpower pool. 

5) The length of planned institutes will, to a great degree, serve 
as a powerful determinant of the number of persons able to take 
port in toe training. 

This assumption was formulated primyhLly on the basis of 
questionnaire returns. Many organizations were reluctant to 
release their employees for longer toan two weeks. It can be noted 
in Table 3 that the figures of 55 percent and 96 percent, for 
evaluation 'and envelopment institutes , respectively, are for an 
institute two weeks in length. The percentages fall off from 
that point? the figrrc becomes about fifty percent for two two-'week 
institutes (a oomrni torn t of four weeks during the year) and drops 
to approx irately thirty-five percent when the ooimvLtment becomes six 
weeks d\iring the year. It would appear that most groups could not 
attend institutes in excess of four weeks during a single year, and 
to at employees in certain categories mi git be able to attend only 
two weeks per year. From oomnants written in on the questionnaire, 
it is also apparent (hat timing of the institutes is critical; there 
are certain peak work periods during the year when many potential 
trainees would not be able to be released from their present employment. 

6) It is assumed that pronouno-md changes roust take place in the 
context in which, and the contingencies under which, persons 
trained in the institutes an employed in order that, their now 
skills can be effectively utilized. 

Ihe assumption being made here, and one for which we ha \*2 only 
tire roost subjective type of data (primarily inforral observations) , 
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is that it is not sinply the lad; of skills in development and evaluation 
that is apparent in rrost of the educational o rgaai 2 a ti ons in this 
country. Rather the assunption is forwarded that there a tv well- 
defined and pcwerful inhibitory forces that keep even well-learned 
skills from readying fruitful e>qpression in educational settings. 

As an exanrple, consider the frustrated director of research or the 
recently- trained Title- I evaluator who funds that his job description 
and responsibilities in no way reflect the training he has received; 
there is likely a gulf between the perceptions of a superintendent 
and an evaluator as tb what the evaluator's ro.le and responsibilities 
are. 

In other words, the assunpticn is that intensive institutes 
that further develop and perfect evaluation and dewloprnent skills 
still will not be sufficient to insure that persons thus trained will 
function effectively in their home environment . Thus, the design 
staff be 1 lews that it is necessary and important to instruct other 
personnel in school and educational administrative hierardiies 
(1) on the proper role and effective use of excision -oriented evaluators 
and (?) on the pro par and effective use of developers. It is proposed 
that such simple rental instruction 1>3 undertaken as one phase of the 
operation of the training program; it would appear to be a vital 
service function. 

7) The as svnpti on is node that research is needed to determine the 
effectiveness of various possible follow- models at the oon- 
conclusion of institute training. 

It is norrally assum’d, and probably rightly so, that even intensive 
institute training doL*s not have lcnq-ian^ substantial effects 
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unless it is aca:>npjuiied by follow-up experiences. Obviously, 
varying degrees cf follow-up could be under taken; variations in 
intensity are nonrally nut died by i:onoOTninant variations in the 
amojnt of resources required for the follcw-up. It seems nit isor able 
that there exists, sorrertxne arrong the infinite variations of possible 
follcw-ip experiences, a n optimal procedure, or at least procedures 
that are most cost effective. 

This Icing the case, it is proposed in the evaluation section of 
tins proposal (Section VIII) that: questions of this nature be 
answered experimentally. A random group of trainees v;ould be given 
extensive follow-up on tho job, a second group would te called bade 
to the training site for additional sessions, a third group would 
be worked with only by phone and rreil, and a final group would receive 
tire traditional treatment (t hat is, essentially no follc^-up at all) . 

It is felt that such a procedure, if used i • the first eighteen months 
or so of the training, would permit the answering of questions that 
would then allow structuring around the most effective follow-up 
procedure daring tlie remainder of tho operational period cf trie 
training program. 

8) It is osSlLtm! th it there exists extensive needs tor the developimnt 
of training mite rials in liie aioas of evaluation and development. 

A recent RFP from the U.S. office of Education requested new 
ideas for training material packages. The AEFA task force has likewise 
conducted a search for promising training practices and materials. 

All indications point to a dearth of effective materials; possibly 
rn^ro important, there seems to };>o a pronounced shortage of viable 
training alternati ves or novel ideas in the urea. 
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program, neecla art? fell’ primarily in two areas. First, a need exists 
for simulation materials than might uc* used effectively with training 
in evaluation aid development. A second apparent need is for the 
development of training pa kage rodulos for the nod el of intensive 
training institutes proposed. 'Ihe need for simulation materials and 
nrate rials of the latter type, the training package modules, is elaborated 
upon in Section VII. of this technical pap r. Also, by referring 
to Section IV.B. of thi s paper, the reader can got a better idea of 
how the training nodules will permit, to a large extent, individualizing 
and tailoring the training institutes to the specific needs of 
part ioular groups . 

9) It is assured that the training program design, though innovative, 
must also bo feasible. 

A very real trade* off exists when one tries co design a training 
program that abounds with originality rjid novelty, and yet at the 
same time must be feasible enough that it has substantial operational 
validity. The Colorado consortium has attempted to proceed at the 
fulcrum between these two extremes. It clearly is difficult to 
brook out of existing training rodos that he-jo existed for several 
decades in the schools and universities. It is also difficult to 
propose d i anges so sweeping that their feasibility and practicality 
are questioned by the essentially conservative; educational 
establishment. Nevertheless , it is felt that a reasonable balance 
has Ixxjn readied between the need for innovativeness and the r.eod 
for feasibility. Further, it is uecogni :;._d that these two qualities 
arc not inooirpatiblo within tJ»o sain: t rut rung program. 




17 



16 



'I ho assiurpt ions outlined in this section were influential in 
effecting the training program that was designed. ihc assunptions 
plus multiple sources of data irpu u led to the establishment of 
nurerous objectives for the training program, and these objectives 
are explicated in the following section of this paper. 
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III. Object i ves of the Tr ain ing Program. 



Goals for the training program can be stated at at 
least three levels of generality. At the most general 
level are broad goals of training vhich relate to the types 
of person to be recruited , a general description of the 
activities in which they will participate , and job descriptions 
and descriptions of work settings for trainees. At a greater 
level of specificity are complexes of skills \hidi are to bj 
imparted through the training program. At the rrost specific 
level are behavioral statements of i ns t n-eti ona 1 objectives 
vdd ch guide the to aching- learning process on a day-to-day basis. 
Bro ad Go als 

At tli is level of generality v*e do not distinguish between 
goals for train ing evaluators and goals for training development 
specialists, The broad goals of the p' "gr*m are to attract 
oorrpotent individuals .into the areas of educational evaluation 
and development and train them to assume responsible roles es 
evaluators and dove] opers in research aid developirent centers, 
regional laboratories, sdiool districts, curriculum projects, 
state departments of education, Title ill projects, and similar 
agencies . 

Ihe goals of the PJi.D training program are to produce pro- 
fessionals capable of (a^ establishing and teaching evaluation 
and development programs and coiu^scs at tlie undergraduate and 
graduate lewis in colleges and universities, (b) advising 
master’s and doctoral studencs pursuing l dvanoed degrees in 
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evaluation and development , (c) conducting basic and applied 
research on rnethodoLxjies of evaluation and development, (d) con- 
sulting with educational acy?ncies on those activities, (e) assuming 
leadership roles in professional organizations concerned with 
evaluation and development, and (f) maintaining an active interest 
in pursuing their own further education in the methodologies of 
evaluation and educational development. Such individuals would 
likely take positions within tire academic world. Only half of 
the Iversons trained at the doctoral level u T ould be trained 
towards such a goal, 'the other half would be trained towards 
the goal of ’working in such settings as school districts, state 
departments of education. regional laboratories, research and 
development oenbers, special projects , or private industry. The 
orthosis in their training would be on the administration and or- 
ganization of educational evaluation and development, consultation 
on these activities, and acquisition of the skills necessary to 
perform evali<ation and development successfully. 

Broad goals of training at the master’s level are to provide 
the non-supervisory manpower force for various agencies involved 
in evaluation and development. Persons trained at the rraster’s 
levol would net. )x* expected to assume major responsibi lity for 
the activities of several other professionals engaged in evaluation 
and development. >>phasis jr* training at the master’s level will 
be on inparting specific skills to trainees which can be coordin- 
ated in a team effort in evaluation or development. 

Short-term intensive training institutes will airn at upgrading 
skills of persons already on the job or providing skills to persons 
seeking a redefinition of role in a school district, or other 
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education-related ag onuy. Emphasis will lx? given to training 
such persons to be perexptive and critical consumers of consulta- 
tion and advice front evaluation and development specialists. It 
is expected that persons so trained will be able to carry out 
educational evaluation and development an a limited Level with the 
assistance and counsel of persons trained at a rrore craprchensivn 
and higher lc\ral. 

C onplexes of Skil ls 

At a more specific level, one ce in identify groups of skills 
which will be in parted through tire training programs in both 
evaluation and development: these are presented 1x2 low. 

E valu ation. In this .section ten conplexes of skills important 
to the successful pursuit of educational evaluation are identified. 

1 ) Budgeting and managing hurrvin and material resources. 



faceted endeavor. In nost instances it entails a financial budge c. 
exocxjdi .j tiiat of nany educational research projects. The activi- 
ties of many technicians must be directed. It is essential that 
those who would engage in educational evaluation be trained in 
sene of the techniques of project management and financing. Ih is 
skill anea will receive heavy emphasis in the doctoral program. 

At the master's training level, only those persons identified as 
potential directors of evaluation projects will receive similar 
training in this regard. For the bulk of trainees at the mister's 
level and these participating in intensive institutes, this ann 
will not bo opphasi zed. 



'lire typical educational evaluation is a large multi- 
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2) Identifying, at appropriate levels of generality, the goals 
of the program to be evaluated. 

An inportant first step in any evaluation will typically 
be eliciting, from responsible parties, the goals and objectives 
toward which a program is directed. Identification of these 
goals must be more than a perfunctory attejrpt at soliciting 
verbal goal statements frcen program personnel . When done properly, 
it entails some cf the most sophisticated technology of survey 
research and interviewing. At all thioo levels of training, 
emphasis will )jO given to the process by whim goal statements 
can be identified in written documents, verbal intx previous, and 
program plans. 

3) Assessing the value of program goals. 

In an evaluation, the goals of the program must not be 
accepted at face value, but must be regarded as elements of the 
program requiring direct evaluation. In sere instances the 
justification of goals must o one from empirical research in educa- 
tion or tlie social sciences. As an example, a program aiming at 
die inculcation of reading readiness skills in five -year- old 
children iirrie< Irately raises the evaluative question of the facili- 

tativu effect on leading of the attainment of those skills identi- 
fied as "loading readiness . " Ihe evaluator must know hw to 
scordi for a justification of these goals in enpirical research 
on reading instruction. He ray tind, for exanple, that the case 
for reading readiness has never been adequately escobl ishrd 
empirical research. Hence , lie ray legitimately raise a question 
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of the justification of the program objective. In other instances, 
tiie evaluator nviy have to turn to nononyji ri cal , nonJxhnvioral 
di.sci.pi i nos uudi as philosophy rind taw in smoking Lo eval vib,* 
program goals . A sdioei system may set out :is an objective to 
teach all diildren in the school the Christian e.hic. f Ihe evalu- 
ator must be sensitive to, and should raise the issue (either 
privately with program personnel or publicly in his report) , of 
the legal and philosophical issues concerned with the separation 
of churdi and state in the United States. Sudi goal evaluation 
requires oroad education in the social sciences, philosophy, 
low, etc., end responsiveness to qn^iiion s of value hidi <^re broader 
than those one can hope to investigate in the span of a single 
3 1 udy , 

4) Trcjis luting broad objectives into specific observable 
objectives . 

General goal sta tenants niuit be operationalized into 
specific statements of objectives. c Ihe onus of making this trans- 
lation lies clearly with the evaluator, who possesses the technical 
skilL for doing so, and not on pregram personnel, to whom the 
language oi o;v rationalization and bell ardor ism is foreign arid un- 
familiar. At all levels of training enphosis musr be gi\*jn to 
tJio technology of behavioral statements of instructional objectives. 
One nuv draw ujxin a large body of publ Lshed literature in pursuing 
this goal and provide frequent practical experiences in the trans- 
lation process. 

5) Identifying standards oi norms for judging worth. 
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Thu nuasureinencs and observations taken in an evaluation 
caiviot be translated into judgments of worth without standards or 
norms. The formality of standards or norms nvay wry greatly, 
but never the less a standard of worth is implied whenever judgment 
of worth is derivoa from an observation. Standards may be either 
internal or external to a particular evaluation. External stan- 
dards are represented by collateral data with which observations 
and measurements are aon pared in deriving evaluations and judgments 
of worth. For example, a school system ; my desire racial balance 
in its schools and have attempted to achieve this balance through 
various means. Iho obsc ovation that / 3 pu rcent of all Negro 
pupi Ls attend schools wnicii have less than 25 percent of the student 
body Negro is on ebser vat ion not yielding immediate evaluaciw 
meaning, since no standard for judging this degree of racial 
mixing exists. A standard for judgment could be fou i external 
to the evaluation. In data from Cole-nans* s Equality ri Educational 
opportuni ty it might be found that in a representative soirple 
of school districts across the nation, '35 percent of Negro pupils 
attend scfrxds which are 30 percent Negro, by bringing external 
data to hoar on the ooservut J ons firm on evaluation study, it can 
bo seen that a mudi more sat is factor/ racial mix was achieved 
wi thin tiie senool district in question. 

Train! ng in the use of col lateral data to derive external 
standards lor judging value involves familiarization with data 
sources relevant to education. Trainers must do come aware of 
various agencies and the data collection end reporting services 
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they render. They must ih i ix? trained in the critical 
skills necessary to evaluate properly credibility and 
reliability of e'beir il data, 

In many instances standards for judging worth are 
to be found internally within an evaluation. The whole 
area of comparative experimental design ip a means of 
establishing internal standards by which the worth of certain 
activities can be judged. One program is pitted against 
an alternative program, and the worth of the former is 
Treasured vis -a- vis the outcomes of tea lav ir-r . The study 
of tiie principles of comparative e; pjri:u ntai design teen 
becorres relevant to the problem of establishing norms for 
judging worth in evaluation studios. Thus, it must receive 
appropriate eirphasis in attempts to irpart skills in this 
general area. 

6) Monitoring programs to detect deviations from 
design or specified procedures. 

It is important to knw what one evaluates. 

It is insufficient to accept neie labels when one has invested 
large portions of time aid money in the observation and 
judgment of outcomes. It is necessary that a program be? 
trenitored through site visitations, interview technigucs, 
survey research methods , classroom observation scheduloa, etc. . 
so that the evaluator is clearly aware of the degree U> which 
a program being evaluated was rrade operational. 
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7) Selecting or u'voloping and using valid techniques 
of mvLsnmarrent. to yield i ul'oimU ion on out nomas. 



outccrnes. It is crucial Hi at the proper outcomes be validly 
treasured. Objective valid data on program perforrnance is the 
sine^ejua non of any j us t i f i ab le evaluation. The evaluator 
must have skill in selecting and evaluating those measurement 
techniques that will reveal objective data on outcomes. He irust 
knew when a measurement technique threatens to inisrepresent 
a set of behaviors. Trainees nvst be presided with the 
cri tical ?k i j Is necessc »y to 3 udge the re iiabi 1 1 ty , valid * ty , 
and practicality of standardised and ad hoe measurement 
techniques. They must be able to construct and evaluate 
cognitive aiu affective Treasures for formative and sumrative 
purposes . 

8 ; Krtvloying appropri ate techniques of data analysis. 



..uva of statistical data analysis. He must have a clear 
unde rstandin j of the fuiicin; rental' : a variety of da ta- 
analy t i c technique:. . He i.Bust b ole to use standard 
packages of conputer programs .such as im BIMD scries), 
altho’igh pnxjrajimung skills per se are net essential. 

0) Making reoomrendatiens as a result of tne evaluation. 



end with the collection, analysis, and reporting of data. 



The worth of an educational program lies in its 



The? evaluator must Iaj brr. adly k new 1 edge able in t he 



r lho evaluator’s responsibility to evaluate does not 
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The data do not speak for themselves, 'Hie perceptive evaluator 
acquires a valuable perspective on the educational program 
being evaluated through his Jong and intimate association 
with it. To fulfill his total evaluative responsibility, he 
must make the subtle and personal inferential leaps from 
those data he has gathered and those results he has observed 
to his personal re corn amda t ions for the conduct or conti lua.noe 
of a program, Thi • is a skill not easily taught. Ihe activity 
draws upon accumulated e>perience, wisdom, and judgment of 
tlie evaluator. 

10) Wri ting the evaluation report. 

br^fting the report of an evaluation for tlie relevant 
audience is an activity quite unlike this writing of a report 
cf a research project. The u valuator is topically corrmurii eating 
to in audience which ‘does not share his perspective, his 
grasp of technical topics, nor his interest in technical 
details. The responsibility to oorimciicate findings rest s 
nor . heavily with the evaluator than the researcher. ihe 
evaluator has to Lam to adopt non ted mi Co 1 language. He 
mist refrain iron' cver-rcli auce on tabular presentation of 
da ha analyses . He must avoid discursive oomrontary on test 
validity and it liability aid other topics which nis audience 
will not find central to their ounce ins. At this stage 
of the evaluation endeavor, the evaluator will play a role 
much none akin to the journalist than the scientist. 
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levolopnnnt . 'lho skills of tlio txlucntionnl ^V'volopnrut 
socialist overlap at nuny points with those* of the educational 
evaluator. Hoover, there are inportant areas of difference. 

The following list of eleven skill areas is a delineation 
of many of the oonpetencies required in a broad range of 
educational development projects. 

1) Drawing on research results in planning devo lepirenta 1 
activities. 

The developer will usually take his cues for particular 
developmental activities from published educational research . 
Research reports will never prescribe dowlopnneiital activities; 
rather tliey inspire certain products and procedures. Developers 
must be sufficiently famdliar and capable of reading the 
research literature that they can draw upon relevant findings 
in laying out the broad design of developmental activities . 

Research findings can never do more than suggest variables 
which may be maaipulatod in the production of educational 
products; hut in order for tire 'developer bo make rraximum 
use of research findings r he must be capable of interpreting 
the research literature. Hence, developers must be trained 
mi the structure of the academic disciplines to levant to 
e duca tior i a 1 ac: t i vi tie s . 

2) Conceptualizing systems r their elements, and interrelations 
among those elements. 

Skills in special systems analysis are critical in the 
planning and implementation of educational developments . 
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Developers must ijo capable of seeing single developme 'tal 
activities as parts of larger ongoing educational systems 
if the fruits of developmental activities aie to be inplemented 
and utilized to some good end. 

3) Specifying desired performance outcomes of instruction. 

The general concerns of administrator's are the broad 
notivation for undertaking new educational developments. 

Seldom are these concerns stated in specific performance 
outcomes, before the educational developer is brought into 
the operation. Developers must be capable of translating 
the broad goals for an educational program, hold by program 
personnel, into the specific observable outcomes whicn are 
to be expected from the educational product or program. 

Thus, training in the Ix^haviocal statement of objectives 
is crucial for developers at all levels. 

4) Identifying alternative instructional media and 
ted uniques. 

Educational development involves the manipulation 
of i nstrumcnts to product desired outcomes. Hie media or 
ins trumen tali tics available to educators must be carefully 
studied by dovr ; ape i s , Tne range of £t 3 1 creative means to 
various ends can be fully enplcyed. Developers must be well 
wised in traditional audio- visuc.1 rrodia of instruction, 
organ i z ati ona 1 pi ans , er. d o ther i * s t rumen ta lit Les tl - roix ^ \ 
which < les i red ends of schooling tee achieved. 



5) 



Determining appropriate sequences of topics in 
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instiuction. 
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there die a^nplexity of institutional segments 
raises the problem of alreniative ordering of elements to 
achiew desired goals, the dew leper must possess skills 
in logical task analysis .and empirical coif innation 
tediniques required to produce optimal orderings of topics. 

6) Composing effective oral and written forms of 
i ns tr ucti on a 1 ccn m micati oris . 

lire heart of the deve lopnental activity ourrp rises 
tile created acts through which die instrumentalities of 
instruction are used to bring about the desired objective. 
Developers must haw t ho skills required to compose narrative 
or visual segments of instruction abound whidi the teadiing- 
learning process will center, Editorial skills, writing 
skills, skills in the creation of sequences of programmed 
instruction, etc., arc the specific capabilities one can 
elaborate in this carp lex of skills. 

7) Selecting or devising appropriate techniques for 
measuring outcomes. 

It is usually necessary in developmental activities 
to devise new puper-and-po.noj 1 measures of learning to assess 
the specific objectives of instructional, sequences. Developers 
require careful and intensive training in vrthods of test 
construction. 

8) Designing and managing initial laboratory tests of 
cfcvoloprrenfal tediniques and materials . 
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9) Designing and managing field tryouts and tests. 

The new era of educational development differs 
frem earlier, less successful attempts in that enpirical 
data on product performance is systematically collected and 
fed back into repeated cycles in the development phase. 

The success of the endeavor often depends upon careful field 
tryouts in which data for revision are collected. Developers 
must possess skills in experimental design, test administration, 
and data analysis. 

10) Deporting evaluation of outcomes from laboratory 
and field trials. 

Skill rust be acquired in identifying sources of 
data which bear on the critical development questions 
concerning which aspects of products or programs must be 
revised in order to nuet performance standards. Once the 
relevant data are identified, great skill must be exercised 
in presenting data to development project personnel in 
coup rehens iblo and usable form. 

11) Interpreting evaluation findings. 

The dew lope r must be capable of specifying needed 
revisions, materials, and programs, from the study of field 
and laboratory trials. He must have a critical attituck? 
toward evidence and te able to separate valid frt ocu invalid 
findings. Furtherrrore , he must be able to prescribe revisions 
in products from evidence of success and failure obtained 
in field and laboratory trials. 



o 

ERIC 



31 



30 



It is important to note that tee field as a whole lacks 
traditional materials and arrangements for teaching many 
of these skills, ergo the critical importance of internships. 
Sp ecific Bella vioral Objectives. 

Coordinated v/ith each oorplex of skills are sets of 
instructional activities. Each instructional activity breaks 
down further into se\'eral specific behavioral objectives. 

At this final level of specificity exist literally hundreds 
of statements of outcomes desired from instruction in tee 
different types of training urogram. It is impossible in 
this limited space to enumerate all instructional objective/:. 
Such objectives are in the process oi lacing identified arid 
ultimately will be identified for ail areas of the training 
curriculum. As an illustration of objectives at this lewl, 
we present belcw the state, nent of behavioral objectives 
for cno skill ooirplex within the training program. Ihese 
dojcctrves are related to skills in comparative experimental 
design and data analysis. Ihey arc: illustrative of the numerous 
instr:ctional objectives which can be stated in all other 
instructional areas. 

Del ia vi o ra] ^ ( objectiv es for instruct^, t he design and 
iSii la lysis of comparative _o x periments . 

Upon oonpletion of instruction the learner vill lx? able to 

1) Distinguish comparative experiments , associatio^al 
studies, and status studies and describe the type of functional 
ml atioaships that result from each. 

2) Distinguish between internal and external validity 
and identify the threats to each as discussed by Campbell and 
Stanley. 
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3) Identify the r j >.periinenbal unit 5 a an experiment and 
distinguish this unit from the unit of statistical analysis. 

4) Determine the appropriate sanple size needed to 
obtain a specified level of power in a given experiment and 
oerrpute the power of tests previously conducted. 

5) Explain iiw randomization and blocking function to 
minimize bias and how blocking further results in increased 
experimental precision. 



6} Distinguish between, jrain and nested classifi cat ions 
and te tween fixed, random, finite, and mixed me bis. 

7) State the assumptions underlying a given analysis and 
describe the effects of failure* to rreet a given assumption. 

8) Write the linear model and determine expected mean 
squares and degrees of freedom for a given experiment ; 
ocipute suns of squares r mean squares, and appropriate 

l r -ratios. 

9) Describe the operations run ..a red to set up the 
following experimental designs: 

a) ran dan i zed groins. 

b) randomized blocks. 

c) repeated measures designs. 

d) Latin square designs. 

e) balanced incomplete blocks. 

£) analysis of covariance . 

10) Grap ! i and interpret two and three f rotor interactions. 

11) Compute cstirrutes of variance opponents for random 
classification analyres of variance. 

12; Dist inguish lx* tween plained aid ej< j>ns t facto conparisons 
between levels of a factor. 

13) Select and oonpi'to the preferred multiple oonparison 
procedure for planned and jx>st hoc contrasts. 



Training pnxirams at the doctor T s and .raster ’s level will 
have essentially the sane broad goii.3 and strive to achieve 
essentially tiie same cor.pl ex? s of evaluation and development 
skills, ihe two program will differ in the depth to which 
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in.s t ruction is purstrjd on each of these v-ojectives. Instruction 
at the doctoral level is intended to be broader and penetrating 
and will pursue each topic in greater detail. At the naster's 
level some objectives will be pursued less deeply with less 
enphasis on theory. However, in the catalog of skills 
sought, there will be substantial overlap between training 
at the doctoral ota. raster's levels jn evaluation and development. 
Intensive training institutes will differ from the long-term 
nvistjcr 1 s and doctoral programs hath in the coverage of topics 
and the depth to which they are investigated. Instructional 
objectives for institutes will be selected to meet the unique 
needs of trainees in these programs An intensive training 
institute may emphasize only three or four skill oonplexe: 
and not pursue these studies to the J.jpth that they would . 
pursued at the raster's anvd doctoral levels. 

It is important to note that the Objectives Section 
is sernv/hat traditional in its orientation t ward broad g 
complexes of skills, and specific behavioral objectives, 
to h_» over If joked r e important skills in broad humanist ‘ 
for vxvmple, skills in interpej sonal relationships dura* . 
evil nation and development, skills in task-oriented one 
etc. These typos; of skill, by their very nature , are di 
to explicate and evaluate and have not lx? on explicitly s 
in this proposal. Nevertheless, the program staff is r- 
o gni /ant of tlio important?/! of Jus general area <md 
provide on instructional eir.d moment within which su i. 
skills will lx? nurtured arid divvlopc?d. 
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IV . Elements of the Trai ning Pro gram 

Th o salient elements of tho training programs designed are 
presented be lew. It can bo noted that much of the training foliar 
directly from the objectives stated in the previous section of this 
paper, (Hcwewir, attainment of certain of the objectives will 
require internship experiences, team activities, and interaction between 
staff and students or among students.) Considered below axe both 
tlie graduate degree training programs and the intensive training 
inst i t utes . 

A , G raduate Le vel Train! ng in Evaluation an d Devel opment 

Most of the formal academic work for all four cbgrees (rasters 
in evaluation and development; doctorate in evaluation and development) 
will be conducted at the University cf Colorado, Boulder campus. 

Each of the four programs will have unique elements, while elements of 
aurmuna) ity will also be in evidence. Several i ns tructi on al techniques, 
for exanpie, will be oormnn to all four programs. 'Ihey are enumerated 
IX) l (J\\ 

1) Class roan instruc tion: In each of the training programs, it is 

envisioned that approximately 40 percent or slightly less of the training 
experienoe will be conducted in formal classroom situations. The 
essential curricular experiences in the training of educational evaluators 
and developers are outlined later in this section. In rrany cases, 
now courses have been constructed, in order to be highly relevant for 
the program. Stbstantial amounts of knowledge and techniques from 
the social sciences have toon incorporated into the courses. Add it tonal ly, 
tr.utx.on have options i.o elect irv.lv> *ndv! it stub/ i: ; areas of high interest 
to them. 
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2) In terdisciplinary ox ientati on and seminar s: It is anticipated that 

each electoral trainee will be oposod to interdisciplinary' work of at 
least two types:; seminars or graduate level courses in other relevant 
disciplines; and advanced seminars in educational evaluation and 
development conducted by a team of scholars from other disciplines. 
Masters students will take part in a two-phase seminar in evaluation 
(or development) , 

Trie University of Colorado is fortunate to have an active program 
of interdisciplinary social-science research pursued by the Institute 
of Behavioral Science. In addition, research trainees in the 
graduate research t mining program of tlie Laboratory of Educational 
Resear di receive instruction in "general social systems" from Kenneth 
lb Bool ding, cne of the three founders of the General Systems Society, 
in tlie Department of Eca-ioru.es. 

3) Semina r s with peersa Tnese prob lem-or iented ncncredit seminars 

will \xj held weekly, Knphosis will be on using actual problem situations 
in evaluation or development, situations posed by practitioners 
attending the seminar seeking assistance. In many cases, trainees 
from all programs (including the current Title IV researdi program) 
will attend the same weekly seminar, thereby promoting faci litative 
fjorri'unication , training, intense tutoring, and bite rat rtion r ung trainees 
in evaluation, develqp/xnt, and researdi . 

4 ) Simulated materials, exercises, and prob lems in evaiuation and 

d evelopmen t : Sere materials are already prepared and will be 

used by trainees individually or in small gin ups. Other materials 
will tx developed in the cos irse of the project (s*.v desvri utions of 
Si relations I, II, ana Til in Section VII). 

3G 
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5) Intems hlp/p racticum ex^xrion ce : Such activities will te a key 

elenent in the training programs, amounting U> <spproximii oly 50 percent 
of the trainee's tiirw and effort. Internships will he offered in tevelop- 
rrexit and various evaluation roles. Receiving institutions will not 
provide any financial support for the intern , nor wi3 3. receiving sites 
be paid by the project to house an intern. It is felt that the value 
of the work done by the intern for the agency should balance the costs 
of ^t^jervising and housing the intern. Hcvever, at certain development 
sites, actual instruction will be provided (Seminar in Educational 
Development) ; when the seminar ir> conducted by a development staff 
neuter, the necessary release time for him wi 11 be paid for by the 
project. .Specific internship sites are outlined later in this section. 

In some cases, it will be desirable and possible to provide trainees 
cith more than one practician experience. In addition, all trainees 
will be exposed to many problem situations in the field and eroouraged 
to formulate appropriate tevalcprrental (or evaluative) procedures, 
ihe^r 'as ters Prog rani in Evaluatio n 

I he master's degree program in educational evaluation is seen 
as a full year experience (two senes ters and a sunnier session) . 

Stolents selected will te of the hi guest possi ble quality (see Section VI) . 
Hie bulk of tlie formal course w*k will be corrpletcd the first semester 
in residence, leaving the rexruining six or seven months primarily 
for extensive field experiences as an intern as 1 as electives when 
feasible, hie diagram below depicts the sequence of 0001*503 in the 
piqgra ■. 
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Fall Songster 



Spring S eries tor 



Summer Se ssxon 



1) Research Methods and 
Applied Statistics 



5) Seminar in Educational 
Evaluation I 



> II 



2) Educational Evaluation 



6) Supervised Internship 
in Educational 
Evaluation I 



3) Instrument: Djsign and 
Development 



» II 



4) Educational Measurement 



Hie content and credit hours associated with each of the courses 
and experiences is designated below. (It might be noted that 1, 3, 

5/ and 6 are all new courses.) 

1) Research Methods and Applied Statistics: 5 horn's. 

- Critical examination of current educational research 
1 i te raturj . 

- i^ole of empirical methods in education. 

- Measures of central tendency, variability, and correlation. 

- Inferential statistics. 

- Experimental and quasi ^xperiirental designs aid analyses. 

* Controlling sources of internal and external invalidity. 

* Factorial designs and interactions. 

* Analyses of covariance. 

- Survey research methods. 

Conputer as a research tool (lab) . 

- Flew charting (PERT, etc.). 

2) Educational Evaluation: 3 hours. 

- Evaluation and research. 

- Evaluation .and measurement . 

- r Ihe ethics of evaluation. 

- Evaluation models. 

- Simulated evaluation activities. 

- Instruments in educational evaluation. 

- Systems analysis. 

- Cost-benefit analysis. 

- Behavioral objectives. 
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3) Instrument Deign and Development: 3 hours. 

- Development and use of cognitive measures. 

- Development and use of attitudinal measures . 

* Q-sort . 

* Semantic differential. 

* Likert scales. 

- Development and use of sociometric measures. 

- Conceptualization and use of unobtrusive measures. 

- Reliability assessment methods. 

- Validity determination. 

- Item sampling. 

- Criterion- referenced measurement. 

- Self report u-nnures. 

4 ) Educational Measurement : 3 ] lours . 

- Evaluation and measurement. 

- Measurement in sdiool settings. 

- Tteadier-made tests . 

- Reliability. 

- Validity. 

- Types of items. 

- Standardized tests. 

* Achievement, 

* General mental ability. 

* Special abilities (creativity , etc.) . 

* Special aptitudes. 

* Interest and personality surveys. 

- Informal measurement tediniqies. 

- Projective bests. 

5) Seminar in Educational Evaluation, I and II: Eadi 2 hours. 

r lhe focus of these seminars wi!3 1 x 2 on applied problems 
in evaluation. Scheduled to coincide with the period that 
the student is an intern, it is expected that rany presentations 
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will be by die train. ?es themselves on problems encountered 
at the internship site. Other sessions will provide exposure 
to persons filling roles in evaluation and development, roles 
which the trainee might select upon completion of his raster's 
program, 

6) Internship in Educational Evaluation, 7. and II: Each 2-6 hours. 

It is obvious that the primary focus of the last six 
or seven months' training will be the internship, accounting lor 
the bulk of tlie trainee's program during that time period. 

In some cases, a trainee can spend the full internship at one 
site and get experience working on various skills. On other 
cases, it will probably be wise to novo students to a second 
site after severaL months at the first. Extensive supervision 
will be provided by the internship site staff (and sometimes 
instruction when it can be arranged) ; soire supervision will 
also be conducted by the project staff. 

A wide range of internship sites is avail able for evaluation 
students at tie master's level . Among them are: 

- John F. Kennedy Oiild Development Center, Denver, Colorado. 

- Denver Public Schools, Denver, Colorado. 

- Northern Colorado Educational Boa* 1 of Cooperative 
Services, Boulder, Colorado. 

- Colorado Department of Education, Denver , Colorado, 

- Ford Foundation Field Office, Denver, Colorado. 

- Office of Education Regional Office?, Denver, Colo ratio 

- 1 >ub] ic school di s L r j cl s vi 11# j n f i f t y mi le radi us . 
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